feat: add support for custom judges via evaluation metric key #86
Conversation
knfreemLD
changed the title
| Default Judge-specific AI Config with required evaluation metric key. | ||
| """ | ||
| messages: Optional[List[LDMessage]] = None | ||
| # Deprecated: evaluation_metric_key is used instead |
There was a problem hiding this comment.
Since we are sub 1.0 release as long as we can guarantee the api is always returning the new single key we should be able to just drop this and do a breaking change. They only thing that really makes this breaking is people will need to update their defaults if they defined it. If you want to drop it now update the PR to be "feat!: ".
I won't block if you want to leave this in for a little while but it likely isn't necessary. The real question is how long do we want to continue sending the old values in the API as that is what will break older SDKs.
There was a problem hiding this comment.
For now we want to make sure this is non-breaking, but soon we're going to remove "legacy" support. For keeping this change as minimal and safe as possible I'd err on the side of caution and keep it in for the time being.
Requirements
Related issues
https://launchdarkly.atlassian.net/browse/REL-11511
See tech spec at https://docs.google.com/document/d/1lzYwQqCcTzN_2zkxJZDfJtgUcEJ4jbpx0KSsJ2bRENw/edit?tab=t.0#heading=h.69bdm7karsxh
Describe the solution you've provided
Updating the SDK to check the AI Config's evaluationMetricKey property which now exists. Also added missing tests from previous implementation, and fallback to the original evaluationMetricKeys list.
Describe alternatives you've considered
Provide a clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context about the pull request here.
Note
Implements single-key judge evaluation with backward compatibility and comprehensive tests.
evaluationMetricKey(deprecatedevaluationMetricKeys), updatingAIJudgeConfig(Default)serializationLDAIClient.__evaluatenow returns the rawvariation;judge_configextractsevaluationMetricKeywith fallback to first inevaluationMetricKeysJudgeupdated to validate and parse a single metric;EvaluationSchemaBuilderbuilds a single-key structured schema; minor cleanup of unused imports/commentsWritten by Cursor Bugbot for commit c6d086a. This will update automatically on new commits. Configure here.